-
Notifications
You must be signed in to change notification settings - Fork 425
OAK-11267: Upgrade Azure SDK V8 to V12 for oak-blob-azure #2409
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: trunk
Are you sure you want to change the base?
Conversation
Commit-Check ❌ |
| try { | ||
| CloudBlobContainer container = Utils.getBlobContainer(connectionString, containerName); | ||
| container.createIfNotExists(); | ||
| boolean useSDK12 = Boolean.parseBoolean(azProps.getProperty(AZURE_SDK_12_ENABLED, "false")); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please use SystemPropertySupplier
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
where is SystemPropertySupplier being used?
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| #Empty config to trigger default setup | ||
| # Licensed to the Apache Software Foundation (ASF) under one or more |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's the chamge here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
must have been on the fly change at commit time performed by git, I see a special endline char was added, does not appear in my changes
oak-run-elastic/pom.xml
Outdated
| 105 MB: Azure updates | ||
| 107 MB: RDB/Tomcat (OAK-10752) | ||
| --> | ||
| <max.jar.size>119000000</max.jar.size> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please add the reason for the increase here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
azure sdk v12 libraries
* OAK-11299: Missing Segments With Oak Run Segment Copy * add ArchiveIndexComparator * use ArchiveIndexComparator in segment migrator * shuffle archive list before sorting * ArchiveIndexComparator singleton --------- Co-authored-by: Lucas Weitzendorf <lweitzendorf@adobe.com>
reschke
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the choice of the SDK depends on a system property, how do we get test coverage for the old SDK?
|
ok, exports look good. somebody else needs to check whether we need all embeds, and whether they use the right (latest?) version. |
reschke
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the choice of the SDK depends on a system property, how do we get test coverage for the old SDK?
| private AzureBlobStoreBackend azureBlobStoreBackend; | ||
| private AbstractAzureBlobStoreBackend azureBlobStoreBackend; | ||
|
|
||
| private final boolean useAzureSdkV12 = Boolean.getBoolean("blob.azure.v12.enabled"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the choice of the SDK depends on a system property, how do we get test coverage for the old SDK?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I need to check how the original author tested that.
|
| private AzureBlobStoreBackend azureBlobStoreBackend; | ||
| private AbstractAzureBlobStoreBackend azureBlobStoreBackend; | ||
|
|
||
| private final boolean useAzureSdkV12 = SystemPropertySupplier.create("blob.azure.v12.enabled", true).get(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
default value must be false
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
| props.putAll(filtered); | ||
| } | ||
|
|
||
| props.setProperty("blob.azure.v12.enabled", "true"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
default value must be false
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks to not be fixed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ahh, is in test, so it's ok
* OAK-11299: Missing Segments With Oak Run Segment Copy (#1892) * OAK-11299: Missing Segments With Oak Run Segment Copy * add ArchiveIndexComparator * use ArchiveIndexComparator in segment migrator * shuffle archive list before sorting * ArchiveIndexComparator singleton --------- Co-authored-by: Lucas Weitzendorf <lweitzendorf@adobe.com> * Fix a typo in builtin_nodetypes.cnd: @peop -> @prop (#2373) * OAK-11834 - Cleanups to reduce changeset in OAK-11814 (#2411) * OAK-11801 : removed Guava's fluent iterable with Apache's (#2408) * OAK-117141: oak-standalone: update dependencies (#2400) done. * Clarify support of SNS in Oak Only reading is supported via JCR API * OAK-11824 lucene analyzer factory should be able to load indirect filters (#2413) * OAK-9447: Upgrade Mongo java driver to 5.2 (#2226) * OAK-9447: Upgrade Mongo java driver to 5.2 (#1787) * Conflicts resolved * Reordered imports like original classes and fixed oak-run about new mongodb-driver-sync * Upgraded com.mongodb and org.bson versions in Import-Package * Restored original formatting * Restored original import ordering --------- Co-authored-by: raffaega <raffaega@CI00298583> Co-authored-by: raffaega <raffaega@host.docker.internal> Co-authored-by: Raffaele Gambelli <r.gambelli@gmail.com> Co-authored-by: Raffaele Gambelli <raffaele.gambelli@cegeka.com> * OAK-9447: cleanup formatting changes * OAK-9447: Fix build and removed unwanted changes due to conflicts. * OAK-9447: Removed some warnings and reverted some unwanted changes * OAK-9447: Some imports ordering and use ping command to check Mongo availability * OAK-9447: reverted some changes in imports and keep functionality. * OAK-9447: revert some formatting changes. * OAK-9447: avoid formatting changes * OAK-9447: Restore functionality of hasWriteConcern. * OAK-9447: Some empty lines cleaning and fix ReplicaSetDefaultWriteConcernIT * OAK-9447: remove empty line * OAK-9447: Bump to version 5.2.1, since 5.2.0 requires an OSGi annotation import. * OAK-9447: Fix import-packages. * OAK-9447: Restored previous annotations, as they are not always enforced. * OAK-9447: Applied PR suggestions. * OAK-9447: Add back mongoURI * OAK-9447: Update MongoFullGcNodeBinSumBsonSizeTest to new Mongo Driver. * OAK-9447: Solve casting issue in MongoFullGcNodeBinSumBsonSizeTest. * OAK-9447: Updated Number casting on numeric values returned by Mongo. This is something that changed in Mongo Java Driver 5.2: https://www.mongodb.com/docs/drivers/java/sync/current/reference/upgrade/#version-5.0-breaking-changes * OAK-9447: Fix type in MongoFullGcNodeBinSumBsonSizeTest. --------- Co-authored-by: raffaega <raffaega@CI00298583> Co-authored-by: raffaega <raffaega@host.docker.internal> Co-authored-by: Raffaele Gambelli <r.gambelli@gmail.com> Co-authored-by: Raffaele Gambelli <raffaele.gambelli@cegeka.com> * OAK-11831: Update Oak trunk to Jackrabbit 2.22.2 (#2417) * OAK-11831: Update Oak trunk to Jackrabbit 2.22.2 - fix version numbers in composite ITs * OAK-11825 - Improve thread monitoring in the OutOfBandIndexer (#2416) * OAK-11774: Remove usage of Guava's Splitter - fix split on regex for Windows (#2420) * OAK-11834: Add delay to getDefaultWriteConcern to avoid UNKNOWN ClusterType (#2422) * OAK-11834: Add delay to getDefaultWriteConcern to avoid UNKNOWN ClusterType * OAK-11834: Cover Sharded clusterTypes too * OAK-11814 - Improve error handling in ElasticResultRowAsyncIterator to deal with timeouts of both slow Elastic queries and slow readers. (#2410) * OAK-10790: FullTextBinaryTextExtractor fails to extract text from csv (#2414) * OAK-10790: FullTextBinaryTextExtractor fails to extract text from csv * OAK-10790: FullTextBinaryTextExtractor fails to extract text from csv- updated comment (#2424) * OAK-11837: Update commons-csv version to 1.14.1 (#2423) * OAK-10674 Use Oak's Bloom filter (#2412) * OAK-11787 ElasticRegexPropertyIndexTest.regexPropertyWithoutFlattened * OAK-10674 Use Oak's Bloom filter * OAK-10674 Use Oak's Bloom filter * OAK-10674 Use Oak's Bloom filter * OAK-10674 Use Oak's Bloom filter * OAK-10674 Use Oak's Bloom filter * Revert "OAK-10674 Use Oak's Bloom filter (#2412)" This reverts commit c438a3f. * OAK-11840: Release Oak 1.84.0 - Candidate Release Notes (#2428) * OAK-11835: Expose all Mongo connection pool parameters (#2426) * OAK-11835: Added OSGi configuration for all Mongo Connection Pool parameters * OAK-11835: Remove unneeded connection to Mongo and move the logging to the actual connection. * OAK-11835: Add new properties to the builder * OAK-11835: Avoid "magic number" in Mongo lease timeout connections. * OAK-11835: Create MongoClientSettings with all the parameters * OAK-11835: Add detailed connection details logs * OAK-11835: Imports cleanup * OAK-11835: This now should be obsolete. * OAK-11835: Add default constants for Mongo connection pool. * OAK-11835: Apply default configuration in getDefaultBuilder * OAK-11835: Added some test cases * OAK-11835: Fix default leaseSocketTimeout. * OAK-11835: Add all parameters to log message * OAK-11835: Fix lease socket timeout connection * OAK-11835: Use millis instead of ms in parameter definitions * OAK-11835: Change all the properties to Millis to keep consistency * OAK-11839 : added throttler to throttle the document store based on external factor set inside settings collection (#2429) * OAK-11389: added throttler to throttle the document store based on external factor set inside settings collection - sonar complaints (#2431) * OAK-11843: Improve diagnostics for NPE in SecureNodeBuilder (#2433) * OAK-11843: Improve diagnostics for NPE in SecureNodeBuilder (#2434) Added null checks. * OAK-11845: Make SecureNodeBuilder thread safe (#2435) done. * Revert "OAK-11845: Make SecureNodeBuilder thread safe (#2435)" This reverts commit bf3f78f. * OAK-11847 - AzureJournalFile fails to rotate journal (#2438) * OAK-11847 - AzureJournalFile fails to rotate journal * OAK-11847 - AzureJournalFile fails to rotate journal * OAK-11840: Release Oak 1.84.0 - Candidate Release Notes * [maven-release-plugin] rollback the release of jackrabbit-oak-1.82.0 * Revert "[maven-release-plugin] rollback the release of jackrabbit-oak-1.82.0" This reverts commit 124381d. * [maven-release-plugin] prepare release jackrabbit-oak-1.84.0 * [maven-release-plugin] prepare for next development iteration * OAK-11808: Segment graph may need to be recomputed (#2393) * OAK-11808: SegmentGraph class * address review comments * address further comments --------- Co-authored-by: Lucas Weitzendorf <lweitzendorf@adobe.com> * OAK-11850 - Use MutableInt to compute facets to avoid boxing/unboxing between int and Integer (#2441) * OAK-11849 - Make facet evaluation timeout configurable. (#2440) * OAK-11842: Copy Bloom Filter implementation from oak-run-commons to oak-commons for re-use (#2432) * OAK-11842: Copy Bloom Filter implementation from oak-run-commons to oak-commons for re-use * OAK-11842: Copy Bloom Filter implementation from oak-run-commons to oak-commons for re-use - remove unintended changes * OAK-11842 Bloom Filter: improve javadocs --------- Co-authored-by: Thomas Mueller <thomasm@apache.org> * OAK-11852: Use new Bloom filter implementation in document store (#2443) * OAK-11852: Use new Bloom filter implementation in document store * OAK-11842: Copy Bloom Filter implementation from oak-run-commons to oak-commons for re-use (#2432) * OAK-11842: Copy Bloom Filter implementation from oak-run-commons to oak-commons for re-use * OAK-11842: Copy Bloom Filter implementation from oak-run-commons to oak-commons for re-use - remove unintended changes * OAK-11842 Bloom Filter: improve javadocs --------- Co-authored-by: Thomas Mueller <thomasm@apache.org> --------- Co-authored-by: Thomas Mueller <thomasm@apache.org> * OAK-11852: add missing test class (#2444) * OAK-11855: update oak-blob to use oak-commons bloom filter (#2445) * OAK-11832 : replaced Guava's TreeTraversal with OAK commons (#2421) * OAK-11832 : replaced Guava's TreeTraversal with OAK commons * OAK-11832 : replaced Guava's TreeTraversal with OAK commons * Update oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/Traverser.java Co-authored-by: Julian Reschke <reschke@apache.org> * Update oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/Traverser.java Co-authored-by: Julian Reschke <reschke@apache.org> * OAK-11832 : added random data generator test to verify guava & oak travsersal codes. * Update oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/Traverser.java Co-authored-by: Julian Reschke <reschke@apache.org> --------- Co-authored-by: Julian Reschke <reschke@apache.org> * OAK-11856: oak-run-commons: remove usage of old Oak bloom filter, use oak-commons instead (#2446) * OAK-10856: oak-run-commons: remove usage of old Oak bloom filter, use oak-commons instead * OAK-10856: oak-run-commons: remove usage of old Oak bloom filter, use oak-commons instead - undo change to search-elastic * OAK-11840: Release Oak 1.84.0 - oak-doc* versions * OAK-11858: Remove remains of Oak bloom filter in oak-run-commons (#2449) * OAK-11859: update Tomcat dependency to 9.0.108 (#2451) * OAK-11851 - Log performance metrics of statistical facets and simplify logic (#2442) * OAK-11832 : replaced Guava's TreeTraversal with OAK Commons - fix Import in testcase * OAK-11224: remove guava dependencies from POMs - oak-store-composite * OAK-11846 oak-segment-azure - recovery will recover from .bak (#2452) * OAK-11846 oak-segment-azure - recovery will recover from .bak * OAK-11833 : replaced Guava's Traverser with OAK commons (#2447) * OAK-11833 : replaced Guava's Traverser with OAK commons * OAK-11833 : throw NPE is root node is null * OAK-11833 : added NPE test for Guava to verify new behaviour with old one * OAK-11857 - Improve path handling in oak-segment-azure * OAK-11866 Support for datastore files without length (#2458) * OAK-11866 Support for datastore files without length * OAK-11866 Support for datastore files without length * OAK-11836 : removed Guava's AbstractIterator (#2456) * OAK-11862: missing facets for multi-valued fields (#2455) * OAK-11864 : added new ALL_ORPHANS FullGC mode for deleting only ALL_ORPHANS node i.e. without any Empty Props (#2459) * OAK-9586: SysViewImportHandler does not support expanded names in sv:name attributes Fixed. * OAK-11861 - segment store initialization should run in parallel (#2454) * OAK-11656: Remove usage of Guava Suppliers.memoize (#2457) * OAK-11656: Remove usage of Guava Suppliers.memoize - move commons.function to commons.internal.function (#2463) * OAK-11832 : moved Traverser to internal.graph package and marked as i… (#2462) * OAK-11832 : moved Traverser to internal.graph package and marked as internal * OAK-11832 : exported internal.graph package * OAK-11869 : removed usage of Guava's hashing with apache commons-codec (#2461) * OAK-11869 : removed usage of Guava's hashing with apache commons-codec * OAK-11869 : fixed issues with data from ByteBuffer and storing of long values in byteBuffer * OAK-11869 : moved code to utils to avoid copy/paste * OAK-11869 : renamed method from hash to hashMurmur32 * OAK-11870: Allow passing fullGCGeneration parameter to RevisionsCommand (#2465) * OAK-11300: update package dependencies wrt primitives (#2468) * OAK-11300: shaded-guava: remove packages that are not used (anymore) (ongoing) - primitives * Revert "OAK-11300: shaded-guava: remove packages that are not used (anymore) (ongoing) - primitives" This reverts commit c94f4ca. * OAK-11300: shaded-guava: remove packages that are not used (anymore) (ongoing) - note that primitives are used by concurrent * OAK-11300: shaded-guava: remove packages that are not used (anymore) (ongoing) - note that primitives are used by concurrent * OAK-11878 ClusterNodeStoreLock LOCK_TIMEOUT is too large (#2469) * OAK-11875: Introduced test case for empty properties/Split documents issue (#2470) * OAK-11864 : fixed 2 failing unit cases (#2473) * OAK-11875 : added unit cases for EmptyProps and split document issue (#2472) * OAK-11300: shaded-guava: remove packages that are not used (anymore) … (#2474) * OAK-9212 AzureArchiveManage.listArchives() should not delete segments (#2467) * OAK-9212 do not delete blobs when doing AzureArchiveManagerV8#listArchives and AzureArchiveManager#listArchives * OAK-9212 remove unused imports * OAK-9212 mock listing blobs during initialisation * OAK-9212 upload deleted marker when the archive is being deleted, and use the same marker to exclude the archive when invoking listArchives * OAK-9212 rename method * OAK-9212 change field visibility * OAK-9212 upload deleted marker when the archive is being deleted, and use the same marker to exclude the archive when invoking listArchives (Azure SDK v12) * OAK-9212 use readOnly when instantiating SegmentArchiveManager * OAK-9212 changed exported version for org.apache.jackrabbit.oak.segment.spi.persistence.split * OAK-9212 use constant * OAK-9212 throw UnsupportedOperationException when in read-only mode * OAK-9212 throw UnsupportedOperationException when in read-only mode * OAK-9212 added test testListArchivesInReadOnlyModeWithPartiallyDeletedArchive * OAK-9212 added test testListArchivesInReadWriteModeWithPartiallyDeletedArchive --------- Co-authored-by: Miroslav Smiljanic <miroslav@apache.com> * Revert "OAK-9212 AzureArchiveManage.listArchives() should not delete segments…" (#2479) This reverts commit 2e63881. * OAK-11861 - segment store initialization should run in parallel (#2478) - move ForkJoinUtils to o.a.j.o.commons.internal.concurrent - align naming (execute/invoke/submit) in ForkJoinUtils with the naming in Java's Executor API * OAK-11883: Remove usage of Guava collect.Range (#2476) * OAK-11881: Remove usage of Guava Maps.map() (#2475) * OAK-11884: AzureArchiveManage.listArchives() should delete segments only if write acces is allowed (#2480) * OAK-11885: Remove usage of Guava collect.ListMultiMap (#2481) * OAK-9897 - SplitPersistence: FileReaper cannot finish cleanup (#665) * OAK-11887: oak-it: exclude *db artefacts (#2483) * OAK-11886 : replaced Guavs's Futures.immediateFuture with JDK completableFuture (#2482) * OAK-11888: Remove usage of Guava ThreadBuilderFactory (#2484) * OAK-11225: remove unused Guava related imports (ongoing) * OAK-11224: remove guava dependencies from POMs (ongoing) - oak-security-spi (#2485) * OAK-11224: remove guava dependencies from POMs (ongoing) - guava now only test dependency in oak-core (#2486) * OAK-11224: shaded guava is only a test dependency in commons, adjust core-spi to re-add the dependency (#2488) * OAK-11129 Improve Lucene documentation (#2491) * OAK-11875 : avoid removing props/docs if they have empty split props (#2489) * OAK-11875 : avoid removing props/docs if they have empty split props * OAK-11875 : added split props that caused FullGC to skip the document * Revert "OAK-11224: shaded guava is only a test dependency in commons, adjust …" (#2492) This reverts commit 1178a98. * OAK-11889 : replaced Guavas's Futures.successfulAsList with JDK comple… (#2487) * OAK-11886 : replaced Guavs's Futures.successfulAsList with JDK completableFuture * OAK-11889 : incorporatd review comments * OAK-11889 : fixed for sonar issue * OAK-11890 : removed usage of Guava's Futures.allAsList (#2493) * OAK-11894 resolve the IdentifierManager lazily (#2494) * OAK-11894 resolve the IdentifierManager lazily * OAK-11893 : removed usage of Guava's ListenableFuture (#2495) * OAK-11891 : added stats for documents that were skipped due to presence of empty split props (#2496) * OAK-11897 Remove exports from oak-segment-azure (#2499) * OAK-11900: SplitPersistenceTest: disable three tests on Windows platform (#2500) * OAK-11896 : removed usage of Guava's SettableFuture from Test cases (#2497) * OAK-11903 : removed usage of Guava's SettableFuture from Prod and its related test files (#2501) * OAK-11905: Update Mockito dependency to 5.19.0 (#2503) * OAK-11906 : removed usage of ListableFutureTask (#2502) * OAK-11908: Update Easymock dependency to 5.6.0 (#2506) * OAK-11907 : removed Guava's DirectExecutor with oak-commons implementation (#2505) * OAK-11909 : removed Guava's newDirectExecutorService with oak-commons implementation (#2507) * OAK-11902: remove existing blobs in AzureTarWriterTest setup (#2512) Co-authored-by: Lucas Weitzendorf <lweitzendorf@adobe.com> * OAK-11267: Upgrade Azure SDK V8 to V12 for oak-blob-azure --------- Co-authored-by: Lucas Weitzendorf <lweitzendorf@gmail.com> Co-authored-by: Lucas Weitzendorf <lweitzendorf@adobe.com> Co-authored-by: Karol Lewandowski <karol86@gmail.com> Co-authored-by: Nuno Santos <nsantos@adobe.com> Co-authored-by: Rishabh Kumar <rishabhdaim1991@gmail.com> Co-authored-by: mbaedke <manfred.baedke@gmail.com> Co-authored-by: Konrad Windszus <kwin@apache.org> Co-authored-by: Fabrizio Fortino <fabrizio.fortino@gmail.com> Co-authored-by: José Andrés Cordero Benítez <Joscorbe@users.noreply.github.com> Co-authored-by: raffaega <raffaega@CI00298583> Co-authored-by: raffaega <raffaega@host.docker.internal> Co-authored-by: Raffaele Gambelli <r.gambelli@gmail.com> Co-authored-by: Raffaele Gambelli <raffaele.gambelli@cegeka.com> Co-authored-by: Julian Reschke <reschke@apache.org> Co-authored-by: Julian Reschke <julian.reschke@gmx.de> Co-authored-by: Mohit Kataria <mkataria@apache.org> Co-authored-by: Thomas Mueller <thomasm@apache.org> Co-authored-by: Ieran Bogdan <48093736+ierandra@users.noreply.github.com> Co-authored-by: Julian Sedding <jsedding@apache.org> Co-authored-by: horia_poradici <horia_poradici@yahoo.com> Co-authored-by: Miroslav Smiljanic <smmiroslav@gmail.com> Co-authored-by: Miroslav Smiljanic <miroslav@apache.com> Co-authored-by: Jörg Hoh <joerghoh@users.noreply.github.com>
- fixed the property name of azure blob log enabling - obtained property via SystenPropertySupplier
- code review fixes
- code review fixes
* OAK-11972: eliminate uses of org.apache.jackrabbit.core.data.RandomInputStream and refactor test code package locations in oak-commons * OAK-11972: eliminate uses of org.apache.jackrabbit.core.data.RandomInputStream and refactor test code package locations in oak-commons * OAK-11972: eliminate uses of org.apache.jackrabbit.core.data.RandomInputStream and refactor test code package locations in oak-commons - package info * OAK-11972: eliminate uses of org.apache.jackrabbit.core.data.RandomInputStream and refactor test code package locations in oak-commons - package info - Sonar nits * OAK-11931 : fix default for prevNoPropCachePercentage (#2559) * OAK-11892 - Expose hidden mount for elasticsearch indexes in IndexStats (#2490) Co-authored-by: chibulcu <chibulcu@adobe.com> * OAK-11267: Upgrade Azure SDK V8 to V12 for oak-blob-azure * OAK-11964 - refactor parallel compaction inheritance (#2548) * OAK-11972: package-info chmod * OAK-11970 : updated MongoDocker Rule to use Mongo 8 (#2565) * OAK-11971 : updated MongoProcess to use Mongo 8 (#2566) * OAK-11972: remove leftover in oak-segment-tar pom (#2567) * OAK-11267: Upgrade Azure SDK V8 to V12 for oak-blob-azure - some test fixes and cleanup * OAK-11267: Upgrade Azure SDK V8 to V12 for oak-blob-azure - some test fixes and cleanup * OAK-11974: remove usage of jackrabbit-data - NamedThreadFactory (#2568) * OAK-11267: Upgrade Azure SDK V8 to V12 for oak-blob-azure - code review comment fixes * OAK-11267: Upgrade Azure SDK V8 to V12 for oak-blob-azure - code review comment fixes * OAK-11895 - CheckpointCompactor writes to "after" instead of "onto" NodeState (#2549) * OAK-11977 Tree store: BufferOverflowException (#2571) * OAK-11899 - use default value in case config value can not be parsed (#2509) * OAK-11899 - use default value in case config value can not be parsed * OAK-11899 - if config parameter can not be converted, use default value * OAK-11899 - increase org.apache.jackrabbit.oak.spi.security version * OAK-11969 do not check the existence of the tree twice (#2560) * OAK-11267: Upgrade Azure SDK V8 to V12 for oak-blob-azure - code review fixes * OAK-11934 - segment preloading for PersistentCache (#2569) * OAK-11914 : removed usage of Guava's HashBasedTable (#2573) * OAK-11914 : removed usage of Guava's HashBasedTable * OAK-11914 : removed un-necessary usage of computeIfAbsent method * OAK-11911 : exposed DirectExecutor from ExecutorUtils (#2575) * OAK-11910 : exposed DirectExecutor from ExecutorUtils * OAK-11910 : marked DirectExecutor as public again * OAK-11911 : increased minor version * OAK-11267: Upgrade Azure SDK V8 to V12 for oak-blob-azure - code review fixes --------- Co-authored-by: Julian Reschke <reschke@apache.org> Co-authored-by: Julian Reschke <julian.reschke@gmx.de> Co-authored-by: stefan-egli <stefanegli@apache.org> Co-authored-by: chibulcuteanu <paul.chibulcuteanu@gmail.com> Co-authored-by: chibulcu <chibulcu@adobe.com> Co-authored-by: Julian Sedding <jsedding@apache.org> Co-authored-by: Rishabh Kumar <rishabhdaim1991@gmail.com> Co-authored-by: Thomas Mueller <thomasm@apache.org> Co-authored-by: waldoro <waldek.r@gmail.com> Co-authored-by: Jörg Hoh <joerghoh@users.noreply.github.com>
…rade_azure_sdk_12
| File tempFile = fileFactory.createTransientFile(name, ".tmp", null); | ||
| tempPath = tempFile.toPath(); | ||
| // Mark for deletion on JVM exit as a safety measure | ||
| tempFile.deleteOnExit(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is done by the TransientFileFactory and is not needed anymore
| // Prepare headers for the presigned URI | ||
| BlobSasHeaders headers = new BlobSasHeaders() | ||
| .setCacheControl(String.format("private, max-age=%d, immutable", httpDownloadURIExpirySeconds)) | ||
| .setContentType(downloadOptions.getContentTypeHeader()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
null/empty values handled by sdk?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
com.azure.storage.blob.specialized.BlobAsyncClientBase#generateSas(com.azure.storage.blob.sas.BlobServiceSasSignatureValues, com.azure.core.util.Context)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm! There are a couple of minor issues/comments.
jsedding
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
During the build, the following warnings are logged
[WARNING] Manifest org.apache.jackrabbit:oak-blob-cloud-azure:bundle:1.87-SNAPSHOT : Unused Import-Package instructions: [com.microsoft.aad.msal4jextensions*, com.microsoft.aad.msal4jextensions.persistence*]
[WARNING] Manifest org.apache.jackrabbit:oak-blob-cloud-azure:bundle:1.87-SNAPSHOT : Export org.apache.jackrabbit.oak.blob.cloud.azure.blobstorage, has 6, private references [com.azure.core.http, com.azure.core.http.policy, com.azure.storage.blob, com.azure.storage.blob.sas, com.azure.storage.blob.specialized, reactor.core.publisher]
While the first is harmless, the second one needs to be addressed IMHO. In all fairness, the build wasn't clean before either, see below. But this PR makes the problem much bigger. This should be avoided, as it can cause real-world class loading errors that are very difficult to analyze.
[WARNING] Manifest org.apache.jackrabbit:oak-blob-cloud-azure:bundle:1.87-SNAPSHOT : Export org.apache.jackrabbit.oak.blob.cloud.azure.blobstorage, has 2, private references [com.microsoft.azure.storage, com.microsoft.azure.storage.blob]
|
|
||
| public class AzureDataStore extends AbstractSharedCachingDataStore implements ConfigurableDataRecordAccessProvider { | ||
|
|
||
| private static final Logger log = LoggerFactory.getLogger(AzureHttpRequestLoggingPolicy.class); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| private static final Logger log = LoggerFactory.getLogger(AzureHttpRequestLoggingPolicy.class); | |
| private static final Logger log = LoggerFactory.getLogger(AzureDataStore.class); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
| private AzureBlobStoreBackend azureBlobStoreBackend; | ||
| private AbstractAzureBlobStoreBackend azureBlobStoreBackend; | ||
|
|
||
| private static final String AZURE_SDK_12_ENABLED = "azure.sdk.12.enabled"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| private static final String AZURE_SDK_12_ENABLED = "azure.sdk.12.enabled"; | |
| private static final String AZURE_SDK_12_ENABLED = "blob.azure.v12.enabled"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
- code review fixes
# Conflicts: # oak-run-commons/src/main/java/org/apache/jackrabbit/oak/fixture/DataStoreUtils.java
- fix build: osgi package version update for oak api jmx
* OAK-11972: eliminate uses of org.apache.jackrabbit.core.data.RandomInputStream and refactor test code package locations in oak-commons * OAK-11972: eliminate uses of org.apache.jackrabbit.core.data.RandomInputStream and refactor test code package locations in oak-commons * OAK-11972: eliminate uses of org.apache.jackrabbit.core.data.RandomInputStream and refactor test code package locations in oak-commons - package info * OAK-11972: eliminate uses of org.apache.jackrabbit.core.data.RandomInputStream and refactor test code package locations in oak-commons - package info - Sonar nits * OAK-11931 : fix default for prevNoPropCachePercentage (#2559) * OAK-11892 - Expose hidden mount for elasticsearch indexes in IndexStats (#2490) Co-authored-by: chibulcu <chibulcu@adobe.com> * OAK-11267: Upgrade Azure SDK V8 to V12 for oak-blob-azure * OAK-11964 - refactor parallel compaction inheritance (#2548) * OAK-11972: package-info chmod * OAK-11970 : updated MongoDocker Rule to use Mongo 8 (#2565) * OAK-11971 : updated MongoProcess to use Mongo 8 (#2566) * OAK-11972: remove leftover in oak-segment-tar pom (#2567) * OAK-11267: Upgrade Azure SDK V8 to V12 for oak-blob-azure - some test fixes and cleanup * OAK-11267: Upgrade Azure SDK V8 to V12 for oak-blob-azure - some test fixes and cleanup * OAK-11974: remove usage of jackrabbit-data - NamedThreadFactory (#2568) * OAK-11267: Upgrade Azure SDK V8 to V12 for oak-blob-azure - code review comment fixes * OAK-11267: Upgrade Azure SDK V8 to V12 for oak-blob-azure - code review comment fixes * OAK-11895 - CheckpointCompactor writes to "after" instead of "onto" NodeState (#2549) * OAK-11977 Tree store: BufferOverflowException (#2571) * OAK-11899 - use default value in case config value can not be parsed (#2509) * OAK-11899 - use default value in case config value can not be parsed * OAK-11899 - if config parameter can not be converted, use default value * OAK-11899 - increase org.apache.jackrabbit.oak.spi.security version * OAK-11969 do not check the existence of the tree twice (#2560) * OAK-11267: Upgrade Azure SDK V8 to V12 for oak-blob-azure - code review fixes * OAK-11934 - segment preloading for PersistentCache (#2569) * OAK-11914 : removed usage of Guava's HashBasedTable (#2573) * OAK-11914 : removed usage of Guava's HashBasedTable * OAK-11914 : removed un-necessary usage of computeIfAbsent method * OAK-11911 : exposed DirectExecutor from ExecutorUtils (#2575) * OAK-11910 : exposed DirectExecutor from ExecutorUtils * OAK-11910 : marked DirectExecutor as public again * OAK-11911 : increased minor version * OAK-11981: blob-plugins - remove use of TransientFileFactory (#2576) * OAK-11267: Upgrade Azure SDK V8 to V12 for oak-blob-azure - code review fixes * OAK-11983: benchmarks: remove jackrabbit-core dependency (#2577) * OAK-11980 - improve parallelization of I/O during segment-azure initialization (#2574) * OAK-11985 : added MongoServerUnavailableException into transient errors (#2582) * OAK-11982 move the calculation of the WARN limit out of the constructor (#2585) * OAK-11913 : created Forwarding executir service (#2580) * OAK-11935 : updating aws sdk from 1.x to 2.x (#2558) * OAK-11935 : updating aws sdk from 1.x to 2.x * OAK-11935 : added more unit cases * OAK-11935 : added unit cases for S3RequestDecorator * OAK-11935 : fixed issue when running ITs with different encryption mode * OAK-11935 : removed unused imports * OAK-11935 : incorporated review comments * OAK-11935 : added review comments * OAK-11935 : added properties for cross Region access * OAK-11935 : removed additional list for removing keys in case of GCP mode * OAK-11935 : moved out CRUD operations from Utils to S3CrudHelper * OAK-11935 : added unit cases for waitForBucket method * OAK-11935 : make total attemps and delay configurable to use lower values in unit cases * OAK-11935 : remaned S3CrudHelper to S3BackendHelper * OAK-11987 - org.apache.jackrabbit.oak.segment.azure.tool.SegmentStoreMigrator.migrateBinaryRef is missing a null check (#2586) * OAK-11984 Support UserId Change for External Users (#2581) * OAK-11984 Support UserId Change for External Users * Removed unused change * Update oak-auth-external/src/main/java/org/apache/jackrabbit/oak/spi/security/authentication/external/impl/ExternalLoginModule.java Co-authored-by: Alejandro Moratinos <Amoratinos@users.noreply.github.com> * Update oak-auth-external/src/main/java/org/apache/jackrabbit/oak/spi/security/authentication/external/impl/ExternalLoginModule.java Co-authored-by: Alejandro Moratinos <Amoratinos@users.noreply.github.com> * moving constants * Added FF * Added tests for FF * Added debug log --------- Co-authored-by: Alejandro Moratinos <Amoratinos@users.noreply.github.com> Co-authored-by: angela <anchela@adobe.com> * OAK-11936: Allow updating the inference config via JMX (#2525) * OAK-11936: Allow updating the inference config via JMX * OAK-11936: Apache License added to new classes --------- Co-authored-by: marvinw <marvinw@adobe.com> * OAK-11949: Sort union queries without "order-by" by score (#2540) * OAK-11949: Sort Union Queries without order-by by score * OAK-11936: restructure if-clause, move import and one additional test * OAK-11949: use double for Null protection --------- Co-authored-by: marvinw <marvinw@adobe.com> * OAK-11267: Upgrade Azure SDK V8 to V12 for oak-blob-azure - code review fixes * OAK-11267: Upgrade Azure SDK V8 to V12 for oak-blob-azure - fix build: osgi package version update for oak api jmx * OAK-11912 : created DirectExecutorServiuce in oak-commons (#2579) * OAK-11936: Allow updating the inference config via JMX - fix package version * OAK-11936: Allow updating the inference config via JMX - line ends in test case * OAK-11994 remove unused dependency to joda-time (#2593) Co-authored-by: Joerg Hoh <jhoh@adobe.com> * OAK-11997: Log slow Mongo queries in DocumentNodeStore (#2596) * Revert "OAK-11936: Allow updating the inference config via JMX - line ends in test case" This reverts commit dea0956. * Revert "OAK-11936: Allow updating the inference config via JMX - fix package version" This reverts commit 7081c97. * Revert "OAK-11936: Allow updating the inference config via JMX (#2525)" This reverts commit 174dce1. * Reapply "OAK-11936: Allow updating the inference config via JMX (#2525)" This reverts commit acac0de. * Reapply "OAK-11936: Allow updating the inference config via JMX - fix package version" This reverts commit 248d4ec. * Reapply "OAK-11936: Allow updating the inference config via JMX - line ends in test case" This reverts commit 09a603b. --------- Co-authored-by: Julian Reschke <reschke@apache.org> Co-authored-by: Julian Reschke <julian.reschke@gmx.de> Co-authored-by: stefan-egli <stefanegli@apache.org> Co-authored-by: chibulcuteanu <paul.chibulcuteanu@gmail.com> Co-authored-by: chibulcu <chibulcu@adobe.com> Co-authored-by: Julian Sedding <jsedding@apache.org> Co-authored-by: Rishabh Kumar <rishabhdaim1991@gmail.com> Co-authored-by: Thomas Mueller <thomasm@apache.org> Co-authored-by: waldoro <waldek.r@gmail.com> Co-authored-by: Jörg Hoh <joerghoh@users.noreply.github.com> Co-authored-by: Johnson Ho <johnho@adobe.com> Co-authored-by: Nicola Scendoni <nscendoni@adobe.com> Co-authored-by: Alejandro Moratinos <Amoratinos@users.noreply.github.com> Co-authored-by: angela <anchela@adobe.com> Co-authored-by: Marvin <95419378+ChlineSaurus@users.noreply.github.com> Co-authored-by: marvinw <marvinw@adobe.com> Co-authored-by: Joerg Hoh <jhoh@adobe.com> Co-authored-by: José Andrés Cordero Benítez <Joscorbe@users.noreply.github.com>
Upgrade azure sdk 12
| try (InputStream fis = new BufferedInputStream(new FileInputStream(tempFile))) { | ||
| blockBlobClient.upload(fis, tempFile.length(), true); | ||
| } finally { | ||
| tempFile.delete(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Minimally, it should be logged when the return value was false.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
| // and loading the entire stream into memory is too risky | ||
| if (recordLength < 0) { | ||
| LOG.debug("Metadata record length unknown. metadataName={}. Saving to temporary file before upload", name); | ||
| File tempFile = createTempFileFromStream(input, name); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would strongly recommend against using tihs. The TransientFileFactory was created ages ago:
deleteOnExitonly helps when the VM is terminated normally- it attempts to use GC to clean up
- it only hides bugs in the caller; debugging a stream/file leak will be much harder than using regular code
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
- fixed comments lost after merge - fixed code analysis issues
OAK-11267: Upgrade Azure SDK V8 to V12 for oak-blob-azure
|
|
@jsedding @rishabhdaim - could you please review again? |
jsedding
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The warning I was concerned about has gone. Seemingly because the package org.apache.jackrabbit.oak.blob.cloud.azure.blobstorage is no longer exported.
If that is indeed not a problem, that's fine with me.
One more thing that is slightly concerning is the long list of optional imports. Are we confident that all of these imports are actually optional? I.e. that their lack will not result in runtime exceptions?




No description provided.